Research of Decision Tree on YARN Using MapReduce and Spark

Decision tree is one of the most widely used classification methods. For massive data processing, MapReduce is a good choice. Whereas, MapReduce is not suitable for iterative algorithms. The programming model of Spark is proposed as a memory-based framework that is fit for iterative algorithms and interactive data mining. In this paper, C4.5 is implemented on both MapReduce and Spark. The result of each layer of the decision tree can be kept in memory in the implementation on Spark. Through the experiments of C4.5, we observed an improvement of 950% on Spark than on MapReduce when the dataset is small. When the number of lines reached 50 million, Spark still kept an improvement of 73%. We concluded the algorithms and applications applicable for MapReduce and Spark. In the discussion section further experiments were performed to confirm our conclusions.